Computerized Adaptive Testing that Allows for Response Revision: Design and Asymptotic Theory

In Computerized Adaptive Testing (CAT), items are selected in real time and are adjusted to the test-taker’s ability. While CAT has become popular for many measurement tasks, such as educational testing and patient reported outcomes, it has been criticized for not allowing examinees to review and revise their answers. In this work, we propose a novel CAT design that preserves the efficiency of a conventional CAT, but allows test-takers to revise their previous answers at any time during the test. The proposed method relies on a polytomous Item Response model that describes the first response to each item, as well as any subsequent responses to it. Each item is selected in order to maximize the Fisher information of the model at the current ability estimate, which is given by the maximizer of a partial likelihood function. We establish the strong consistency and asymptotic normality of the final ability estimator under minimal conditions on the test-taker’s revision behavior. We present the findings of two simulation studies that illustrate our theoretical results, as well as the behavior of the proposed design in a realistic item pool.